feat(eval): batch-evaluation simulate — each example owns its invoker - #2032

What

Adds batch-evaluation simulate to replay a dataset against a runtime and grade the resulting sessions.

Each dataset example is a self-running object that owns its own invoker (RunContext). invokeDataset builds one invoker per session and passes it into the example's run(). The example performs the invocation; the machine supplies the invoker and collects the result.

The invoker seam

A dataset example knows what to send and how to turn the responses into ground truth, but not how to reach the runtime. Its RunContext owns that runtime-specific invocation:

// src/core/eval/invokeDataset/example/types.ts
export type TurnResult = { text: string };
export type RunContext = { invokeOnce(input: string): Promise<TurnResult> };

export interface Example {
  readonly exampleId: string;
  readonly schemaType: DatasetSchemaType;
  // Replay this example against the runtime `ctx` reaches, return neutral ground truth.
  run(ctx: RunContext): Promise<InlineGroundTruth | undefined>;
}

invokeDataset resolves the runtime once, then builds a distinct invoker for each client-generated session and passes it to that example:

const results = await runExamples(examples, async (example) => {
  const ctx: RunContext = {
    invokeOnce: async (input) => {
      const res = await invokeRuntime(
        deps,
        { /* resolved runtime, session, and rendered payload */ },
        options,
        signal,
      );
      return { text: /* drained response body */ };
    },
  };

  return example.run(ctx);
});

The example decides how many times to call invokeOnce and in what order. Its invoker decides how each call reaches the runtime for that session.

Layout

src/core/eval/invokeDataset/
├── load.ts                 DatasetLoader — pure JSONL parse → shape-classify → new
├── run.ts                  runExamples — bounded-concurrency pool, failure isolation
├── invokeDataset.test.ts   end-to-end golden coverage of the whole path
└── example/
    ├── types.ts            Example interface, RunContext (the invoker), TurnResult
    ├── predefined.ts       PredefinedExample — replays scripted turns, builds ground truth
    └── simulated.ts        SimulatedExample — not shipped; throws at construction

Plus src/core/invokeRuntime.ts, which extracts runtime invocation from runtime.tsx for reuse by both RuntimeClient and invokeDataset, and src/handlers/eval/batch-evaluation/simulate/index.tsx, which composes invokeDataset with startBatchEvaluation.

Testing

• bun run typecheck passes.
• bun test: 1544 pass, 0 fail.
• invokeDataset.test.ts drives the real EvalClient.invokeDataset over a fake AWS layer with golden fixtures. Its snapshot covers created sessions and inline ground truth across every supported variation while exercising the loader, example classes, concurrency pool, template rendering, and runtime invocation.
• simulate.test.tsx snapshots the handler's wrapped sessionMetadata.